This invention relates to a computer system, data management method and computer for processing big data.
Currently, systems for processing large amounts of various types of data, namely, “big data”, are becoming more widespread. In such systems, processing such as obtaining required data from real time data transmitted from a terminal and the like is executed. Such processing is herein referred to as real time data processing.
Further, in such a system, the processing results of the real time data are stored in a database, and past data is analyzed based on those processing results. Such processing is herein referred to as history analysis processing.
Hitherto, because the database storing the processing results is formed of a plurality of disk apparatus, writing the data to the database is slow. Therefore, the processing performance of real time data processing deteriorates due to the delays in the writing of the data. As a result, there has been a problem in that the real time data processing cannot be carried out securely in real time.
As a technology for solving the above-mentioned problem, an in-memory database configured to store all data in a memory enabling quick access is known. However, in a case where all data is stored in a memory, there is a problem in that equipment costs increase due to a large number of memories being required.
Thus, as a system configured to ensure the processing performance of real time data processing and suppress equipment costs, technologies disclosed in JP 2000-331423 A and JP 07-56740 A are known, for example.
In JP 2000-331423 A, it is disclosed that in order to securely record data such as moving images on a disk recording medium in real time, “data from a moving image input unit 4 is accumulated in temporary storage means, and when the fact that a predetermined amount of the data has been accumulated is detected by data accumulated amount detecting means, disk recording/playback processing means records data on the disk recording medium without being controlled by an external system control unit 6.”
In JP 07-56740 A, it is disclosed that in order to inform a user of the results of analyzing, diagnosing, and inferring data representing a state of a monitoring target continuously for a long period without harming the real time nature of the processing, “input data is chronologically stored in an element buffer of a shared memory configured to store the input data as history data. When the stored input data exceeds the upper limit of the storage capacity of the element buffer, a representative value for the element buffer is determined by a noise removing apparatus in conjunction with noise removing processing for removing data uncertainty. Newer representative values are stored in a separate element buffer in a higher level.”